[WIP][VL] Add snappy to gluten columnar shuffle compression codec#11454
[WIP][VL] Add snappy to gluten columnar shuffle compression codec#11454taiyang-li wants to merge 6 commits intoapache:mainfrom
Conversation
taiyang-li
force-pushed
the
support_snappy_codec
branch
from
58f7870 to
3502acc
Compare
taiyang-li
changed the title
seems not related to current changes.
|
zhouyuan
changed the title
|
Have you tested whether snappy can be used with this patch? I wonder if some changes are needed in the cpp code. For example we need to support building the snappy codec here https://github.com/apache/incubator-gluten/blob/main/cpp/core/utils/Compression.cc#L28-L61 Please also enable |
|
Please convert to draft PR or add [WIP] in the title if the PR is not ready yet. Thanks! |
taiyang-li
changed the title
@marin-ma thanks for your remainding. I'll continue the work. |
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
…PY in Compression.cc, enable SNAPPY in velox shuffle tests, allow snappy in VeloxBackendSettings, and set snappy buffer size (32k). Co-Authored-By: Aime <aime@bytedance.com> Change-Id: Ie25a5bd1e4e3f978e4ce386175bb1eb67d841e3d
taiyang-li
force-pushed
the
support_snappy_codec
branch
from
372bad9 to
0213a52
Compare
|
Run Gluten Clickhouse CI on x86 |
3 participants
What changes are proposed in this pull request?
try to add snappy as gluten columnar shuffle compression codec
How was this patch tested?